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ABSTRACT 

IMS information^ cycles from the school to SWRL for processing, returning 
again to the school in the form of summarized reports of pup.il data. 
One segment of this cycle is man age d= by the IMS pupil data base pro- 
cessing program. The pr:>gram generates reports from processed IMS pupil 
test scores and updates the Pupil Data Base Tape at; a central computing 
facility. This paper describes the characteristics of IMS data processed 
during the first seven weeks of computer operation as interpreted and 
logged by the managing program! / 
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IMS VEkSION 3 1971-72 TRYOUT: INITIAL REP6rT ON 
COMSYS 1 AND 2 COMPUTER OPERATIONS 



In the period of January 1 to February 20, 1972, fourteen IMS runs 
have been processed successfully through the IMS pupil data base pro- 
cessing-program. Of thes^, three (21.4%) were wholly concerned with 
maintaining and updating the Pupil'l)ata Base Tape at UCLA's CCN. The 
remaining eleven "Data Runs" were concerned with storing and reporting 
pupil scores (Table 1). ' ' . 



Def initions and Data . f 

The term "Class Identification" is used to denote the six classi- 
fication variables: District, School, Teacher , Program, Grade, and Class 
A "CLASS ID" sheet with such data must precede a set of pupil data when 
scanned. 

A "Data Run" is defined byjits composition: A group of pupil scores 
from one or more classes processed as a unit. The first 11 Data Runs 
contained pupil scores from 23 distinct classes with an average. of 8,5 
and a range of 3 to 16 classes per Run. The size of a given Data Run is 
largely determined by the rate and volume of data being received , an 
^effort to limit the scanner operator's processing time to under 2 hours, 
and a restriction in program processing limiting the number of Data Re- 
cords accepted to less than 500. 

A "Delete" request is made to eliminate a pupil from the Pupil Data 
Base reports, while an "Update" request adds a new student to the Pupil 
Data Base. Seventeen Deletes have been processed through Data Run 14, 
but since many were submitted erroneously due to a misunderstanding of 
the purpose of the Delete Form (see TN-5-72-03), the. average of 1.5 per 
\ 
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Data Run may be expected to decrease. Fourteen Updates were also pro- 
cessed without error. ^ . s * 

A, "Data Record" is a count of logical records excluding those with 
Class, Identification information which provides one measure of data 
volume processed. A Delete or an Update request is made with one Data 
Record. A FYCSP Test is composed of two Data Records, one for each page, 
and a LMS Test page is composed of from one to ten Data Records; each 
record contains one pupil's test scores. 



Data Run Content 




Over 1,000 Criterion Exer<:ises have been processed through Run 14 
averaging 98 per Data Run. An average of 112 Data_ Records have been 
handled per Data Run with a maximum of 217 processed in Run 1. Thus,^ 
the limit on the number of Data ittecords that IMS programs can handle in 
a Run has not yet been any real restriction on the volume of data com- 
posing < a 'Run. 

Tables 2 and 3 show the Class content of each Data 'Run for LMS and 
FYCSP programs, respectively, indicating the classes which had not yet 
submitted data. Individual teachers have submitted data into a mnKinrum 
of six separate Data. Runs. Forty percent of the teachers currently par- 
ticipating in LMS programs and twenty percent of thosp in ^YCSP programs 
have submitted data. ^ 

A State of Control * ^ 

Unexpected circumstances, as well as clerical errdrs, delay the IMS 
data cycle process as they must be traced, explained, and corrected. A 
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variety of problems affecting the IMS process at* the program processing 
Stage are listed under "Conmients" in Table 1 and in Table 4. Some of 
the/problems may be expected to occur periodically and their frequency 
counts, will embody variations inherent in the IMS program that are im- 
possible, or economically infeasible to remove. In a controlled system,^ 
measurements will vary owing to chance causes only arid, therefore, 
approximately satisfy the concept of randomness. When, a State of Control 
is achieved the mass behavior of measurement variation and the limits 
within which repeated measures may be expected to fall is predictable 
through statistical methods. This variation is coritrasted with variation 

due to some assignable and^ remediable determinant, 

/ 

A _ _ _ / 

As expected in a newly implemented processijng system, a State of 
Control has not been reached and ^nan^^f the prbblems detected to date 
stem from assignable causes. Many deficiencies in IMS programs have 
been corrected and improvements introduced. Likewise, many of thj& 
teachers' clerical mistakes were assignable being due to misunderstand- 
ing the required IMS procedure and forms. The apparent decreasing fre- 
quency of errors with Runs discussed later could be, in part^ due to 
eliminating such misunderstandings. However, since few teachers have 
submitted data into over two Data Runs, there is as yet too little data 
to reliably determine the effect of teachers' experience on the fre- 
quency of problems detected, i.e., the proportion of measjarem^nt variance 
assignable to teacher caused errors. 

Processing Data Runs 

A single Data Run may require more than one computer processing run 
if complications occur. Such complications are noted under '^Comments'' 

i 
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in Table 1 and are followed by "(R)" for rerun. Of the 11 Data Runs, 
6 (54.5%) required more than one computer transmission from SWRL's data 
concentrator to the central computing facility. 

On the first attempt to transmit Run 2. data, a 690 Read Err^br 
occurred. Transmission wa^ successful the next day. In Run 4, an il- 
legal Class ID (District, School, Teacher Program, Grade., and Class) 
message and an -illegal Program-or-lJnit-Number message were printed. More 
Illegal Class id's appeared in Runs 5, 6, 7, and 12 and in each case the 
unprocessed class data were= pulled from the Data Run. The cause was 
traced to a misprinting pf the Class identification Sheets such that 
the teacher's coded ID failed to match the Puptl Data Basi^ codes. Only 
in Runs 7 and 12 was it necessary to rescan th6 data wit^^ the problem 
class. In these Runs the error pertained; to the last cVass being , pro- 
cessed anid hence updating of the Pupil Data Base Tape did not terminate 
properly. All the original reports were good and mailed from the first 
Run. The illegal data record in Run 4 was traced to a student's stray 
mark which was corrected. The entire Run was rescahned and= reprocessed 
by the operator althod^h this was repetitive in terms of reports gener- 
ated. A Pupil Data Base Maintenance Program has been designed (see ^ 
TN-5-72-11) to manipulate 'specified data records individually, but it 
was not operable until February 25. 

In Run 11, the software distorted one class unit's reports. The 
RJS operator attempted to reprocess the data in the absence of the IMS 
syster..s programmer. However, a third. Run was necessary with the compli- 
cating class unit data pulled until the program could be modified: to. 



handle that unit. Another programming bug was detected in Run 13 when 
an error supposedly specific to FYCSP data occurred for LMS data. Cor-' 
rected, the Run^was reprocessed. 

The cause of each computer rerun was "assignable" and thus, with the • 
elimination of each cause, the IMS process approaches its State of Control. 



Interpreting Error Frequency Counts 

The errors detected by the IMS programs are only a subset of those 
known to interfere with IMS data processing. Pupil Data being processed 
under ComSys 1 or 2 input modes initially preprocessed in, the Forms Control 
Station (FCS) where many types of forms and procedure errors are checked 
for, corrected, and logged by date and teacher. They are presently being 

cross-classified by Run Number, a task that should lessen as the flow oJE 

v 

IMS data stabilises. 

Some of these errors are distinguishable by type only through cler- 
ical preprocessing and, therefore , data on them is limited to within the 
ComSys 1 and 2 operating systems.. Through February 9, 1972, the different 
types of errors in this category included: 

Missing Class ID Sheet; only tests received 
\ Missing Class ID Sheet; Update Form used for ID 
^lissing Class ID Sheet; Delete Form used as ID 
Missing test date on ID Sheet, 
Original Class Record Sheet received 
Group designation unclear^'^n Class Record Sheet 
Lines skipped on Class Record Sheet 
Criterion Exercises' stapled together 
Messages written on Criterion Exercise Sheets 
Erasures necessary for scanning 
Marks requiring darkening for scanning 
LMS pupil tests received (stimulus sheets) 
One LMS Criterion Exercise Sheet used, per pupil 
New pupil's name entered on LMS Criterion Exercise Sheet 
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Under ComSys 3, the above errors must be classified together as 
records non-acceptable for processing, the only exception being the 
missing test date on a Class ID Sheet. An error check for this omission 
is being added' to the IMS programs which will estimate the date when 
missing. 

Other forms and procedure errors which are noticed in preprocess- 
ing are corrected ^ven though they are checked for in IMS programs. 
Errors corrected in FCS through February 9 as well as by the IMS program. 

include: • ' 

'\ 1 ' . ' 

Pupil List numbers-unmarked 

."Tested" box not marked '\ 

"Delete" box not marked 

Pupil code omitted on FYCSP Criterion Exercise Sheet ^ 
Hie list of error checks made by IMS programs has been continually 
revised as use reveals more appropriate checks to. be made. These will 
be reported in subsequent documentation of detection procedures* However 
some considerations pertinent to earlier documentation remain applicable 

/ 

and are discussed below. 

Two categories of error are determined by cause: (1) errors caused 
outside the processing system by teacher or student (called Teacher 
Errors) and (2) errors caused within the processing system by software 
logic and operators* mistakes (called System Errors). . 

When an error is detected, it is reported, but the data record may 
either be rejected or passed on for further processing depending ori^ the 
tjiature of the error. Thus, there are also two categories of errors^j'dis- 
tinguished as rejectable and non-rejectable. The' bounds for assigning 
errors to these two categories has been modified as data management pro- 
cedures adapt to new information. 

/ 

/ 

/ 



Since the detection of a Rejectable Error halts further processing, 
subsequent errors occurring on that record remain undetected. 

-If errors'are to be counted*, the processing must be completed, 
allowing all errors to be detected before the record is rejected. As 
is, the-errors must be considered in a hierarchy determined by the order 
in which they are considered by the processing program. Only for the 
error detected first is the frequency count f^^ now available a -true 
measure of its frequency of occurrence F^. For the second error detected, 
on a single Data Record, the count f2 is a miniirwm bound of its actual 
frequence F2. A maximum of f^^ + f2 occurrences of the second error is 
possible. Thus, for the nth Rejectable Er'ror^with f^ occurrences, de- ^ 
•tec ted, ^ 

where f^^ are the detected frequencies of error, i=l tV n in order, of the 
errors' consideration by the program, and E^^ is the actual number of 
errors made on the record being processed. 

If the probability of multiple error s^oh^a* recprd^^was sufficiently 

small, f„ could be assumed equal to F^. This assumption appears only 

\ 

tenuously reasonable, since it also ^seems reasonable that the probability 
of making two errors given one error, was made is greater than the average 
probability of making an error. If this effect is large, and we assume 
it is not, treating these frequencies f^ as error counts is hazardous. 
For example, the actual frequencies of Error I (F^.) and Error J (Fj) 
cannot be compared using their detected frequencies f^ and f . since,* 
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"i»here f,- and- fs are the Ith jand Jth Re jectable Errors to be considered.- 
The magnitude -.of overlap in thes,e intervals will becon/e evident upon 
examining the upper i)punds of the frequencies when further data are ^ 
collected. 

Furthermore, if f^^F^^a summing over the different tjrpes of error\ 
does not give a total count of errors made, but rather the total number 
of Data Records that contained one Ipr more errors* It seems at this point 



more reasonable to judge 



the IMS processing system in terms of defective 
records rathet: than error frequencies; for this, the hierachical- system 
of detection is approprifate. 

Errors Detected In The Pupil Data Base Processing Program 

The frequencies of error as detected by IMS programs and summed 
over the 14 Data Runs (exclusive of Invalid Class Identification previ- 
ously discussed), are found in Table 4 listed by teacher. 

Two types of error were detected with substantially greater fre- 
quencies than the others. The most frequent was the .teacher ' s failure 
to mark a pupil as tested on LMS Criterion Exercise when the item 
responses cleirly indicate that he was. The error is consequential only 
for a pupil who obtains a perfect score. In this case," no error message 
is produced, and his score is not recorded, since his entire data record 
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was blank. Although 57 percent of the 63 errors detected through Run 14 
were of this type, aver half of them (21) were made by one teacher in his 
first Data InpuL. He has submitted subsequent datum, and since notified 
has not ^ p^ted this error. However, even adjusting for him does not 
change this error's position relative to the other errors. 

The second most frequently detected error was caused by teachers 
submitting Criterion Exercises for a' pupil after deleting him from the 
Pupil Data Base. The most frequent cause of this mistake was found to 
be teachers who misunderstood tht intention of the delete option and 
were deleting piipils abjsent the day of the test, * 

Teacher 01 01 03 submitted scores in several program units for a^^ 
pupil and finally deleted him. The two errors listed for this, teacher 



under "Delej:e Requests'* revealed a prograiraning deficiency of failing to 
order pupil data within \^ given Data Run by date ahd/br program unit; 
the student was deleted b^fora his-" scores were reaL Temporarily, only 
one unit per class was included in a Data Run whe 
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Deletes were present. 



The IMS programs were moilified: to handle such situations. 



Teacher 01 03 02 did not actually make six errors as indicated by 



the program count. One pupil !s stray pencil mark invalidated the program 



code white another deliberately altered his identification number on one 
page of an FYCSP Criterion Exercise. The latter jcaused Missing-Page-Error 
messages under his real identification and Duplicate-Tdentif i.cation-Error 
messages under a third pupil's identification. 

Over all, the detection of errors has been reasonably infrequent 
and most of those detected have been' assigned to either a programming logic 
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deficiency or a teacher misunderstanding. As the assignable errors are 
traced and eliminated, the efrror detection frequencies do appear to de- 
crease: Thirty-nine of the erro^rs listed in Table 4 were detected in 
the first four Data Runs while only'nlfrerwet-e detected in Kuns 11 

through 14. ' ' - 

I 

I _ t 





IMS DAT/V PROCESSING RECORD 



Table 1 
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Classes , 


Data X) 
Records 
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Deletes 


Updates 


, Tests 
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1/3 
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4 
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Run 2 
- — 1 . 
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Twenty-three of the 93 were distinct Classes 
The average is taken over the eleven Data Ri 



ns^/ rather than. -thje^ total 14. 
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NuiTibers in parenthesis are for attempted but unsucce^ssful computer rirns 
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DATA PROCESSED FROM LMS PROGRAMS 



Table 2 
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Seventeen more teachers ^coded 02 03 04 through 04 01 04 have not submitted pnpil 
data, 

-Double Xs (XX) imply data Erom two different classes with the same Teacher were 
processed in the same Data Run. 
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.DATA PROCESSED FROM FYCSP PROGRAMS 
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Dpuble Xs (XX) imply data from two different classesi^ with the same Teacher^ were processed 
/in the same Run, 



15 



ERIC 



Table 4 



ERRORS BY EACH TEACHER AS DETECTED IN THE 
PUPIL DATA BASE PROCESSING PROGRAM 



District 

School 

Teacher 
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CRITERION EXERCISES 
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2 


2 


2 


''36 


12 


2 


5 


■ 1 


63 


100% 


PERCENT 
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.03 


.03 


.57 


.19 


.03 


.08 i 


.02 1 


100% 





* Error check modified to apply only to FYCSP data after this one error was detected. 
Crossed out cells indicate that the probability of a positive frequency count in that 
cell is zero, the error being specific to FYCSP or LMS data. ,* 

- ' \ ' / 

** Error check added to the PDBPP '^as of 2/18/72. ; 
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